Migrate monotonic counter metrics to `u64_counter!` #6350

goto-bus-stop · 2024-11-27T14:05:52Z

This is part 1 out of... 4 or 5? of a series of work to move over to our new telemetry macros.

This part does the easiest part :), the tracing::info!(monotonic_counter.) macros that should now use u64_counter!(). I used a lot of commits so I could take notes while doing the work, I'll copy them to PR comments so there's no need to look at each commit individually.

I avoided breaking changes to the metrics for now, so this is targeted at 1.x.

Two uses of tracing::info!(monotonic_counter.) remain, these are already being addressed in #6338.

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

Exceptions

Note any exceptions here

Notes

It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩
Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩
Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

Notes: - Fixed a typo in the not found attribute: `persisted_quieries.not_found` -> `persisted_queries.not_found`. - Added description, it would be useful for someone to check it. It reads to me like *every* request is measured?

Notes: - Removes the `apollo_router_deduplicated_subscriptions_total` metric. This is already captured by `apollo.router.operations.subscriptions` in the `subscriptions.deduplicated` attribute. - The `apollo.router.operations.batching` metric appears to use an older style of attribute naming?

Notes: - The description for `apollo_router_skipped_event_count` may not entirely be correct?

Notes: - This combined a log message and a metric: now they are separate.

svc-apollo-docs · 2024-11-27T14:05:56Z

✅ Docs Preview Ready

No new or changed pages found.

github-actions · 2024-11-27T14:06:08Z

@goto-bus-stop, please consider creating a changeset entry in /.changesets/. These instructions describe the process and tooling.

router-perf · 2024-11-27T14:06:24Z

CI performance tests

apollo-router/src/notification.rs

goto-bus-stop · 2024-11-27T14:07:41Z

apollo-router/src/uplink/mod.rs

@@ -210,7 +210,7 @@ where
    Response: Send + 'static + Debug,
    TransformedResponse: Send + 'static + Debug,
 {
-    let query = query_name::<Query>();
+    let query_name = query_name::<Query>();


Just to clarify that it isn't the full query text.

goto-bus-stop · 2024-11-27T14:08:38Z

apollo-router/src/services/subgraph_service.rs

-            mode = %"passthrough",
+        u64_counter!(
+            "apollo_router_deduplicated_subscriptions_total",
+            "Total deduplicated subscription requests (deprecated)",


I marked a few apparently duplicate, or poorly named metrics as deprecated. This one appears to be a less informative duplicate of apollo.router.operations.subscriptions, which has a subscriptions.deduplicated attribute.

Oh! Good catch. It makes sense.

goto-bus-stop · 2024-11-27T14:09:37Z

apollo-router/src/plugins/authorization/mod.rs

-                            monotonic_counter.apollo_require_authentication_failure_count = 1u64,
+                        u64_counter!(
+                            "apollo_require_authentication_failure_count",
+                            "Number of unauthenticated requests (deprecated)",


I marked this as deprecated because the naming doesn't seem to follow convention, but I don't know if we have a proper alternative for this? We have apollo.router.operations.authorization below but I'm not sure it reports the same thing.

I think that's the right thing to do. cc @BrynCooke you probably have more context

bnjjj

On metrics you marked as deprecated in description that could be interesting to also document it as deprecated in docs. We already have a deprecated section. I know it's also part of another ticket but we both work on deprecating different metrics so in order to not forget any of these deprecated metrics I think it's worth documenting it directly

apollo-router/src/notification.rs

bnjjj · 2024-11-28T14:01:09Z

apollo-router/src/plugins/authentication/mod.rs

+        );
+        u64_counter!(
+            "apollo.router.operations.jwt",
+            "Number of requests with JWT authentication",


I'm wondering if we should not add authentication.jwt.failed = false in the future (for 2.0) to be consistent with what sigv4 is doing for example

apollo-router/src/plugins/authentication/subgraph.rs

bnjjj · 2024-11-28T14:02:23Z

apollo-router/src/plugins/authorization/mod.rs

-                            monotonic_counter.apollo_require_authentication_failure_count = 1u64,
+                        u64_counter!(
+                            "apollo_require_authentication_failure_count",
+                            "Number of unauthenticated requests (deprecated)",


I think that's the right thing to do. cc @BrynCooke you probably have more context

bnjjj · 2024-11-28T14:11:01Z

apollo-router/src/plugins/traffic_shaping/timeout/future.rs

-                tracing::info!(monotonic_counter.apollo_router_timeout = 1u64,);
+                u64_counter!(
+                    "apollo_router_timeout",
+                    "Number of timed out client requests",


You can mark it as deprecated

The alternative is to use a custom instrument, right? So that I can document it in the deprecated metrics section

telemetry: instrumentation: instruments: router: http.server.request.duration: # Adding subgraph name, response status code from the router and the operation name attributes: http.response.status_code: true # If status code == 504 then it's a timeout at the router http request level graphql.operation.name: operation_name: string subgraph: # Adding subgraph name, response status code from the subgraph and original operation name from the supergraph http.client.request.duration: attributes: subgraph.name: true http.response.status_code: # If status code == 504 then it's a timeout at the subgraph http request level subgraph_response_status: code

Custom instrumentation is an enterprise only feature, but we can also get the status code without using enterprise features:

telemetry: instrumentation: instruments: default_requirement_level: required

bnjjj · 2024-11-28T14:13:17Z

apollo-router/src/services/router/service.rs

-                        monotonic_counter.apollo.router.graphql_error = 1u64,
+                    u64_counter!(
+                        "apollo.router.graphql_error",
+                        "Number of GraphQL error responses returned by the router",


Suggested change

"Number of GraphQL error responses returned by the router",

"Number of GraphQL error responses returned by the router (DEPRECATED)",

bnjjj · 2024-11-28T14:13:33Z

apollo-router/src/services/router/service.rs

-                    tracing::info!(monotonic_counter.apollo.router.graphql_error = count,);
+                    u64_counter!(
+                        "apollo.router.graphql_error",
+                        "Number of GraphQL error responses returned by the router",


Suggested change

"Number of GraphQL error responses returned by the router",

"Number of GraphQL error responses returned by the router (DEPRECATED)",

Ah really? I thought we quite recently added more of these (though the metric name is definitely not good). Is there a replacement?

telemetry: instrumentation: instruments: router: http.server.request.duration: attributes: http.response.status_code: true graphql.operation.name: operation_name: string # This attribute will be set to true if the response contains graphql errors graphql.errors: # This will be true if it contains graphql errors on_graphql_error: true supergraph: supergraph.response.errors: type: counter value: event_unit description: number of request containing graphql errors and the error extension code unit: req attributes: graphql.operation.name: true errors.code: response_errors: $[0].extensions.code condition: exists: response_errors: $.*.extensions.code

Here is the alternative

bnjjj · 2024-11-28T14:13:46Z

apollo-router/src/services/router/service.rs

-                        code = code
+                    u64_counter!(
+                        "apollo.router.graphql_error",
+                        "Number of GraphQL error responses returned by the router",


Suggested change

"Number of GraphQL error responses returned by the router",

"Number of GraphQL error responses returned by the router (DEPRECATED)",

bnjjj · 2024-11-28T14:15:26Z

apollo-router/src/services/subgraph_service.rs

-            mode = %"passthrough",
+        u64_counter!(
+            "apollo_router_deduplicated_subscriptions_total",
+            "Total deduplicated subscription requests (deprecated)",


Oh! Good catch. It makes sense.

Co-authored-by: Coenen Benjamin <[email protected]>

…custom instrumentation

This reverts commit 7e02584.

goto-bus-stop added 14 commits November 27, 2024 11:57

Use metrics macros in persisted queries layer

304919a

Notes: - Fixed a typo in the not found attribute: `persisted_quieries.not_found` -> `persisted_queries.not_found`. - Added description, it would be useful for someone to check it. It reads to me like *every* request is measured?

Use new metrics macro for counters in graphql::Request

148d9c4

Use new metrics macro for counting defer requests and responses

be452c9

Use new metrics macro for counting uplink fetches

7f744e9

Use new metrics macro for counting cache hits

eb3a8f7

Use new metrics macro for counting authentication uses

a93408d

Use new metrics macros in timeout layer

f3bc20b

Use new metrics macros for reporting graphql error counts

9130dbc

Use new metrics macros for reporting subscription events

551170f

Notes: - The description for `apollo_router_skipped_event_count` may not entirely be correct?

Use new metrics macros for reporting state changes

b5497c6

Notes: - This combined a log message and a metric: now they are separate.

Use new metrics macros for reporting which telemetry system is enabled

69e71b1

Add back the deprecated metrics

4858d24

fmt

fce826c

goto-bus-stop requested review from a team as code owners November 27, 2024 14:05

goto-bus-stop commented Nov 27, 2024

View reviewed changes

apollo-router/src/notification.rs Outdated Show resolved Hide resolved

goto-bus-stop commented Nov 27, 2024

View reviewed changes

goto-bus-stop requested review from BrynCooke and bnjjj November 28, 2024 10:49

Simplify plugin_metrics

3c43714

bnjjj reviewed Nov 28, 2024

View reviewed changes

goto-bus-stop and others added 2 commits November 28, 2024 15:23

Tweak apollo.router.operations.telemetry description

fd39d9b

Co-authored-by: Coenen Benjamin <[email protected]>

docs: list the newly deprecated metrics

0d6e222

goto-bus-stop requested a review from a team as a code owner November 28, 2024 14:36

goto-bus-stop added 2 commits November 28, 2024 15:43

Add a description for subgraph SigV4 signing

01ebdef

Add apollo.router.operations.authorization description

aca6024

goto-bus-stop mentioned this pull request Nov 28, 2024

Migrate histogram metrics to {f,u}64_histogram! #6356

Draft

6 tasks

goto-bus-stop added 5 commits November 28, 2024 16:40

Tweak apollo_router_skipped_event_count description

cdb7940

docs: add migration instructions for metrics deprecated in favour of …

54b62dd

…custom instrumentation

Docs-deprecate apollo.router.graphql_error metric

7e02584

Revert "Docs-deprecate apollo.router.graphql_error metric"

35a24bb

This reverts commit 7e02584.

Update deprecated metrics docs

947cb6d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate monotonic counter metrics to `u64_counter!` #6350

Migrate monotonic counter metrics to `u64_counter!` #6350

goto-bus-stop commented Nov 27, 2024

svc-apollo-docs commented Nov 27, 2024 •

edited

Loading

github-actions bot commented Nov 27, 2024

router-perf bot commented Nov 27, 2024

goto-bus-stop Nov 27, 2024

goto-bus-stop Nov 27, 2024

bnjjj Nov 28, 2024

goto-bus-stop Nov 27, 2024

bnjjj Nov 28, 2024

bnjjj left a comment

bnjjj Nov 28, 2024

bnjjj Nov 28, 2024

bnjjj Nov 28, 2024

goto-bus-stop Nov 28, 2024 •

edited

Loading

bnjjj Nov 28, 2024

goto-bus-stop Nov 29, 2024

bnjjj Nov 28, 2024

bnjjj Nov 28, 2024

goto-bus-stop Nov 28, 2024

bnjjj Nov 28, 2024 •

edited

Loading

bnjjj Nov 28, 2024

bnjjj Nov 28, 2024

	"Number of GraphQL error responses returned by the router",
	"Number of GraphQL error responses returned by the router (DEPRECATED)",

Migrate monotonic counter metrics to u64_counter! #6350

Are you sure you want to change the base?

Migrate monotonic counter metrics to u64_counter! #6350

Conversation

goto-bus-stop commented Nov 27, 2024

Footnotes

svc-apollo-docs commented Nov 27, 2024 • edited Loading

✅ Docs Preview Ready

github-actions bot commented Nov 27, 2024

router-perf bot commented Nov 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnjjj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goto-bus-stop Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnjjj Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Migrate monotonic counter metrics to `u64_counter!` #6350

Migrate monotonic counter metrics to `u64_counter!` #6350

svc-apollo-docs commented Nov 27, 2024 •

edited

Loading

goto-bus-stop Nov 28, 2024 •

edited

Loading

bnjjj Nov 28, 2024 •

edited

Loading